Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(abfs): Register write filesink #11973

Closed
wants to merge 3 commits into from
Closed

feat(abfs): Register write filesink #11973

wants to merge 3 commits into from

Conversation

zhli1142015
Copy link
Contributor

No description provided.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 27, 2024
@netlify Netlify
Copy link

netlify bot commented Dec 27, 2024 •
edited
Loading

✅ Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 4cafcdd
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/6773381d0a013b0008bca1ca

@zhli1142015
Copy link
Contributor Author

When creating an instance of AbfsWriteFile, we need to check if the file exists. Therefore, even when testing only with dwio::common::FileSink::create, we must replace the real client with a mock client as well.

cc @majetideepak

@zhli1142015
Copy link
Contributor Author

UT failure is not related to this PR.

290/398 Test #380: velox_hdfs_file_test .................................................***Failed   65.24 sec
Running main() from /__w/velox/velox/_build/release/_deps/gtest-src/googletest/src/gtest_main.cc
[==========] Running 21 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 20 tests from HdfsFileSystemTest
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: Logging before InitGoogleLogging() is written to STDERR
E20241227 07:04:46.299198 14359 Exceptions.h:66] Line: /__w/velox/velox/velox/connectors/hive/storage_adapters/hdfs/tests/HdfsMiniCluster.cpp:92, Function:addFile, Expression:  Failed to add file to hdfs, exit code: 383, Source: RUNTIME, ErrorCode: INVALID_STATE
unknown file: Failure
C++ exception with description "Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to add file to hdfs, exit code: 383
Retriable: False
Function: addFile
File: /__w/velox/velox/velox/connectors/hive/storage_adapters/hdfs/tests/HdfsMiniCluster.cpp
Line: 92
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems4test15HdfsMiniCluster7addFileENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_
# 4  _ZN18HdfsFileSystemTest14SetUpTestSuiteEv
# 5  _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_9TestSuiteEvEET0_PT_MS4_FS3_vEPKc
# 6  _ZN7testing9TestSuite3RunEv
# 7  _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
# 8  _ZN7testing8UnitTest3RunEv
# 9  main
# 10 __libc_start_call_main
# 11 __libc_start_main
# 12 _start
" thrown in SetUpTestSuite().

majetideepak
majetideepak reviewed Dec 27, 2024
View reviewed changes
Copy link
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhli1142015 Thanks for adding the tests. A couple of comments.

velox/connectors/hive/storage_adapters/abfs/AbfsConfig.cpp Outdated Show resolved Hide resolved
velox/connectors/hive/storage_adapters/abfs/AbfsConfig.h Outdated Show resolved Hide resolved
velox/connectors/hive/storage_adapters/abfs/AbfsConfig.h Outdated
/// Test only.
static void registerFakeWriteFileClient(
std::unique_ptr<AzureDataLakeFileClient> fakeClient) {
fakeWriteClient_ = std::move(fakeClient);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For safety and simplicity, let's check and initialize the singleton testWriteClient_ here.

@majetideepak majetideepak requested a review from JkSelf December 27, 2024 22:35
JkSelf
JkSelf approved these changes Jan 2, 2025
View reviewed changes
Copy link
Collaborator

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for your work.

majetideepak
majetideepak approved these changes Jan 2, 2025
View reviewed changes
Copy link
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @zhli1142015

@majetideepak majetideepak added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Jan 2, 2025
@facebook-github-bot
Copy link
Contributor

@kevinwilfong has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@majetideepak majetideepak changed the title feat: Register write filesink for abfs connector feat(abfs): Register write filesink Jan 2, 2025
@facebook-github-bot
Copy link
Contributor

@kevinwilfong merged this pull request in c324cd8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JkSelf JkSelf JkSelf approved these changes

@majetideepak majetideepak majetideepak approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zhli1142015 @facebook-github-bot @majetideepak @JkSelf